Layer normalization has demonstrated remarkable effectiveness at preventing plasticity loss in continual and reinforcement learning (RL), though the precise reasons for this effectiveness remain mysterious.

We explain the discrepancy by reproducing the Kaplan et al. scaling law on two datasets (OpenWebText2 and RefinedWeb)

This presents aproblem for certifying unseen points asthe ground truth cannot be known. We therefore stipulate that certification must be independent of the true label of the point being certified. Moreover, replacing the ground truth with the predicted label is unsatisfactory,because thepurpose ofgeneralizing totop-k predictions istoconsider cases where anyofthepredictionsinFk(x)maybecorrect. Wewouldthusliketopredict only whenm(S,x) < 0. To accomplish this we create an instrumented model,g, as given by EquationB2. First, by applying (C7), we obtain (C8).

Now we return to the second part of (9). This illustrates how tight the upper bound is. We use a GeForce RTX 2080 Ti GPU for all experiments. Instead, we always carve out a validation set from our training set. Figure 1 The batch size is set to 128, and 1000 fixed images are used for the validation data. Here we provide the raw hypergradients corresponding to the outer optimization shown in Appendices: Figure 1.